[feature](query) support max_visible_partition() syntax - #66187
Conversation
### What problem does this PR solve? Problem Summary: Support a new `max_visible_partition()` syntax after the table name so a query can be automatically pruned to the latest partition that has visible data, without hardcoding the partition name. - RANGE table: prune to the partition with the greatest range upper bound that has visible data. - UNPARTITIONED table: return the single partition if it has data, otherwise empty. - LIST table: no natural "latest" partition, return empty result. - Only OLAP table and materialized view are supported; other table types are rejected explicitly. ### Release note Support `max_visible_partition()` syntax to query the latest visible partition.
|
run buildall |
TPC-H: Total hot run time: 29463 ms |
TPC-DS: Total hot run time: 176586 ms |
ClickBench: Total hot run time: 24.92 s |
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Request changes.
The basic parser-to-binding flow works for the covered RANGE, MTMV, UNPARTITIONED, LIST, and View cases, and the change is otherwise localized. However, the current implementation has blocking correctness/lifecycle gaps:
- data-dependent partition selection becomes stale in server-prepared short-circuit plans and excludes current-transaction subtransaction partitions;
- time travel selects the current maximum partition rather than the maximum at the requested snapshot;
- the empty shortcut suppresses scan-modifier validation and can expose the wrong output schema;
- CTE and
$meta-table early returns silently ignore a clause that is advertised as OLAP/MV-only; - cloud binding performs one serialized version RPC per expired partition under planner table locks.
Critical-checkpoint conclusions:
- Goal and tests: the happy-path regression demonstrates the advertised simple cases, but it does not prove cloud, transaction, prepared-statement, historical-snapshot, modifier-interaction, or early-return behavior; the accepted inline findings give concrete failing plans for those gaps.
- Scope and parallel paths: the code is compact, but validation and semantics are not applied across CTE/meta-table, direct-index, time-travel, change-read, and prepared execution paths.
- Concurrency/performance: shared-nothing metadata iteration is protected by the planner read lock; cloud mode adds O(partitions) synchronous RPCs while that lock is held instead of using the existing batch API.
- Lifecycle: normal one-shot binding is straightforward, but transaction subtransactions and reusable prepared short-circuit plans are not incorporated into the data-dependent selection.
- Error handling and output correctness: empty data currently changes whether invalid modifiers fail and which schema an empty result exposes.
- Compatibility/configuration/persistence/protocol: the keyword is non-reserved and no parser ambiguity, new configuration, persistence format, FE-BE symbol, or storage-format compatibility issue was found.
- Observability: the new modifier is also omitted from the audit SQL digest.
No additional user review focus was supplied, so the complete PR was reviewed. Per the runner instruction this was a static review only; no builds or tests were run. Existing review context contained no prior inline threads. Reviewed head: 1de88be1a05b3e31669f0a4ee630eb3f4c0d0cc2.
| Partition result = null; | ||
| PartitionKey maxUpperBound = null; | ||
| for (Partition partition : nameToPartition.values()) { | ||
| if (partition.getVisibleVersion() <= Partition.PARTITION_INIT_VERSION) { |
There was a problem hiding this comment.
[P1] Batch cloud version lookups before scanning partitions
On a cloud table, every iteration here dispatches through CloudPartition.getVisibleVersion(). Once the cache entry has expired, that is a synchronous batchMode=false meta-service RPC. Binding runs after StatementContext.lock() and the planner releases that table read lock only after planning, so a table with N expired partitions now incurs N sequential network round trips while holding metadata locks. The existing Partition.getVisibleVersions() / CloudPartition.getSnapshotVisibleVersion() path already fetches an aligned version vector in one batch. Please use that batch result to choose the greatest upper bound; otherwise the feature intended to reduce planning overhead can instead scale with N RPC latencies and combine observations from different instants.
| } | ||
| if (type == PartitionType.UNPARTITIONED) { | ||
| for (Partition partition : nameToPartition.values()) { | ||
| if (partition.getVisibleVersion() > Partition.PARTITION_INIT_VERSION) { |
There was a problem hiding this comment.
[P1] Include current-transaction partitions in the max selection
This test only sees committed visibleVersion, but Doris's transaction-load path intentionally treats partitions containing earlier subtransaction writes as readable: PruneEmptyPartition.selectNonEmptyPartitionIdsForTxnLoad() adds them through TransactionEntry.getPartitionSubTxnIds(). By selecting one committed partition here—or returning null before that rewrite runs—the new path removes the higher pending partition from the candidate set. For example, after one insert-select subtransaction writes p2, a later INSERT INTO dst SELECT ... FROM src max_visible_partition() in the same transaction scans older p1 (or an empty relation) instead of p2. Please merge current-transaction partition evidence into the max ordering before deciding the result is empty.
| .getItems().upperEndpoint(); | ||
| if (maxUpperBound == null || upper.compareTo(maxUpperBound) > 0) { | ||
| maxUpperBound = upper; | ||
| result = partition; |
There was a problem hiding this comment.
[P1] Re-resolve the max partition before reusing a prepared point plan
The chosen ID is baked into LogicalOlapScan, and a qualifying point query can then be cached as a server-prepared short-circuit OlapScanNode. ExecuteCommand reuses that node when ShortCircuitQueryContext.isReusable() succeeds, but that check covers only dropped state, schema version, table name, and a session limit—not visible/table/partition versions. Prepare and execute while p1 is the maximum, commit data into higher p2 without changing the schema, then execute for a key in p2: the cached node remains pinned to p1 and returns the wrong result. Please disable short-circuit reuse for this data-dependent modifier or include the selection/data version and force rebinding when it changes.
| } | ||
| // max_visible_partition() with no visible partition -> empty result instead of | ||
| // falling through to a full-table scan (empty partIds otherwise means "all partitions") | ||
| if (unboundRelation.isMaxVisiblePartition() && CollectionUtils.isEmpty(partIds)) { |
There was a problem hiding this comment.
[P1] Validate and shape all scan modifiers before returning empty
This return happens before direct-index lookup/compatibility, manual-tablet marking, PREAGGOPEN/@incr compatibility, and validateTimeTravel(). Consequently empty_t INDEX missing max_visible_partition() succeeds instead of reporting the missing index, and unsupported or malformed time travel is accepted only while the table is empty. For a valid non-base index, this default scan also supplies the base-table output rather than getOutputByIndex(); for @incr it drops the scan params used to shape the row-binlog output. Please resolve and validate the complete requested scan first, then derive the empty relation from that resolved output so analysis and metadata do not depend on whether data currently exists.
| + "Table: %s is not OLAP table", t.getName())); | ||
| } | ||
| if (unboundRelation.isMaxVisiblePartition()) { | ||
| Partition partition = t.getMaxVisiblePartition(); |
There was a problem hiding this comment.
[P1] Resolve the maximum partition at the requested snapshot
The grammar permits FOR VERSION/TIME AS OF immediately before this clause, but this call uses today's partition versions and only later does buildTimeTravelPlan() apply commit_tso <= targetTso to the already restricted scan. With p1 first populated at TSO 100 and higher p2 first populated at TSO 200, FROM t FOR VERSION AS OF 150 max_visible_partition() becomes:
Filter(commit_tso <= 150) -> LogicalOlapScan(partitions=[p2])
and returns empty instead of the rows from p1; the MOW union receives the same wrong partIds. Please either reject this modifier combination or determine non-emptiness/max ordering against the requested snapshot.
| List<String> qualifierWithoutTableName = qualifiedTableName.subList(0, qualifiedTableName.size() - 1); | ||
| // max_visible_partition() only works on OLAP-backed tables; reject others explicitly | ||
| // so the flag is not silently ignored. | ||
| if (unboundRelation.isMaxVisiblePartition()) { |
There was a problem hiding this comment.
[P1] Enforce this restriction before every relation early return
This guard is not reached for two parser-accepted targets. WITH c AS (...) SELECT * FROM c max_visible_partition() returns a LogicalCTEConsumer/recursive work-table reference after rejectScanParamsOnCte() (which does not inspect this flag), so it scans all CTE rows. Likewise a resolved external $ system table returns from handleMetaTable() above this check as a file/TVF relation. Both silently discard semantics that ordinary external tables reject. Please invoke a shared modifier validation before those early returns, or explicitly reject the flag in each path, and add CTE plus native/TVF meta-table negatives.
| private final List<String> partNames; | ||
| private final List<Long> tabletIds; | ||
| private final boolean isTempPart; | ||
| private final boolean isMaxVisiblePartition; |
There was a problem hiding this comment.
[P2] Include this scan modifier in the SQL digest
The new flag is preserved by the copy/equality paths, but UnboundRelation.toDigest() never emits it. LogicalPlanAdapter captures that digest and AuditLogHelper hashes it into SqlDigest, so SELECT ... FROM t and SELECT ... FROM t max_visible_partition() are grouped as the same statement despite radically different scan scope and performance. Other semantic scan modifiers such as INDEX, TABLET, scan params, and snapshots are represented there. Please append a stable MAX_VISIBLE_PARTITION() token when this flag is set and add a digest assertion.
3 participants
What problem does this PR solve?
Problem Summary:
Support a new
max_visible_partition()syntax after the table name so a query can be automatically pruned to the latest partition that has visible data, without hardcoding the partition name.The common workaround
where pt = (select max(pt) from t)requires an extra sub-query, adding overhead in the Plan / Schedule / Execute phases. This syntax lets the engine resolve the target partition during Nereids binding and reuse the existing partition-pruning path, so the change is confined to FE and does not touch the execution layer.PartitionKey.compareTo, type-aware for numbers/dates, not by partition name).LogicalEmptyRelation) is built instead of throwing or falling back to a full scan.Release note
Support
max_visible_partition()syntax to query the latest visible partition.Check List (For Author)
Test
Regression suite:
regression-test/suites/nereids_syntax_p0/max_visible_partition.groovy, covering RANGE pruning, materialized view, View rejection, and UNPARTITIONED / LIST empty-result cases.Manual test on cluster
doris_ad_ttam_my3(tablead_stats_test.new_uid_sketch_creative_hourly): sub-query form 594ms / 4 fragments vsmax_visible_partition()27ms / 2 fragments.Behavior changed:
max_visible_partition()clause after the table name.MAX_VISIBLE_PARTITIONis a non-reserved keyword, so existing SQL is unaffected.Does this need documentation?
Check List (For Reviewer who merge this PR)